An assessment of preclinical removable prosthodontics based on multiple-choice questions: Stakeholders' perceptions.

PURPOSE: Item analysis of multiple-choice questions (MCQs) is an essential tool for identifying items that can be stored, revised, or discarded to build a quality MCQ bank. This study analyzed MCQs based on item analysis to develop a pool of valid and reliable items and investigate stakeholders' perceptions regarding MCQs in a written summative assessment (WSA) based on this item analysis. METHODS: In this descriptive study, 55 questions each from 2016 to 2019 of WSA in preclinical removable prosthodontics for fourth-year undergraduate dentistry students were analyzed for item analysis. Items were categorized according to their difficulty index (DIF I) and discrimination index (DI). Students (2021-2022) were assessed using this question bank. Students' perceptions of and feedback from faculty members concerning this assessment were collected using a questionnaire with a five-point Likert scale. RESULTS: Of 220 items when both indices (DIF I and DI) were combined, 144 (65.5%) were retained in the question bank, 66 (30%) required revision before incorporation into the question bank, and only 10 (4.5%) were discarded. The mean DIF I and DI values were 69% (standard deviation [Std.Dev] = 19) and 0.22 (Std.Dev = 0.16), respectively, for 220 MCQs. The mean scores from the questionnaire for students and feedback from faculty members ranged from 3.50 to 4.04 and from 4 to 5, respectively, indicating that stakeholders tended to agree and strongly agree, respectively, with the proposed statements. CONCLUSION: This study assisted the prosthodontics department in creating a set of prevalidated questions with known difficulty and discrimination capacity.

Development and validity evidence for the resident-led large group teaching assessment instrument in the United States: a methodological study.

PURPOSE: Despite educational mandates to assess resident teaching competence, limited instruments with validity evidence exist for this purpose. Existing instruments do not allow faculty to assess resident-led teaching in a large group format or whether teaching was interactive. This study gathers validity evidence on the use of the Resident-led Large Group Teaching Assessment Instrument (Relate), an instrument used by faculty to assess resident teaching competency. Relate comprises 23 behaviors divided into six elements: learning environment, goals and objectives, content of talk, promotion of understanding and retention, session management, and closure. METHODS: Messick's unified validity framework was used for this study. Investigators used video recordings of resident-led teaching from three pediatric residency programs to develop Relate and a rater guidebook. Faculty were trained on instrument use through frame-of-reference training. Resident teaching at all sites was video-recorded during 2018-2019. Two trained faculty raters assessed each video. Descriptive statistics on performance were obtained. Validity evidence sources include: rater training effect (response process), reliability and variability (internal structure), and impact on Milestones assessment (relations to other variables). RESULTS: Forty-eight videos, from 16 residents, were analyzed. Rater training improved inter-rater reliability from 0.04 to 0.64. The Φ-coefficient reliability was 0.50. There was a significant correlation between overall Relate performance and the pediatric teaching Milestone, r = 0.34, P = .019. CONCLUSION: Relate provides validity evidence with sufficient reliability to measure resident-led large-group teaching competence.

Smart grading: A generative AI-based tool for knowledge-grounded answer evaluation in educational assessments.

Evaluating text-based answers obtained in educational settings or behavioral studies is time-consuming and resource-intensive. Applying novel artificial intelligence tools such as ChatGPT might support the process. Still, currently available implementations do not allow for automated and case-specific evaluations of large numbers of student answers. To counter this limitation, we developed a flexible software and user-friendly web application that enables researchers and educators to use cutting-edge artificial intelligence technologies by providing an interface that combines large language models with options to specify questions of interest, sample solutions, and evaluation instructions for automated answer scoring. We validated the method in an empirical study and found the software with expert ratings to have high reliability. Hence, the present software constitutes a valuable tool to facilitate and enhance text-based answer evaluation.•Generative AI-enhanced software for customizable, case-specific, and automized grading of large amounts of text-based answers.•Open-source software and web application for direct implementation and adaptation.

Uso do Mini-Clinical Evaluation Exercise (Mini-CEX) na residência médica: uma revisão de escopo / Use of Mini-Clinical Evaluation Exercise (Mini-CEX) in medical residency: a scoping review

Resumo Introdução: Com a evolução do ensino médico para currículos baseados em competências, fez-se necessária uma readequação dos currículos e dos métodos de avaliação, com maior enfoque sobre o cenário de prática profissional e, portanto, na utilização de ferramentas como o Mini-Clinical Evaluation Exercise (Mini-CEX). Objetivo: Este estudo teve como objetivo avaliar o uso da estratégia Mini-CEX como método de avaliação nos programas de residência médica. Método: Trata-se de uma revisão de escopo, cuja estratégia de busca realizada no PubMed resultou em 578 artigos. Após aplicar a metodologia do Instituto Joanna Briggs para inclusão e exclusão, foram selecionados 24 estudos transversais. Resultado: Selecionaram-se artigos referentes a estudos realizados entre 1995 e 2021, em diversos continentes, diferentes programas de residência, e cenários ambulatorial, internação e de emergência. O Mini-CEX mostrou-se aplicável no contexto da residência médica, pois trata-se de uma avaliação observacional direta do atendimento realizado pelo médico residente nos diversos cenários de atuação, como ambulatórios, internações e emergências. Trata-se de uma avaliação com tempo de observação variando de dez a 40 minutos e que permite a abordagem de vários aspectos do atendimento médico, como anamnese, exame físico, raciocínio clínico e aconselhamento, além de possibilitar a realização de um feedback sobre o desempenho dos residentes. Conclusão: O Mini-CEX constitui uma ferramenta de fácil aplicabilidade e promove alto grau de satisfação dos envolvidos, podendo ser utilizada de forma rotineira nos programas de residência médica.

Abstract Introduction: With the evolution of medical education towards competency-based curriculum, the need has emerged to reconfigure curriculum and assessment methods, with increased focus on the professional practice setting, thus leading to the utilization of tools such as the mini-CEX (mini-Clinical Evaluation Exam). Objective: To evaluate the use of the mini-CEX strategy as an assessment method in medical residency programs. Method: This is a scoping review, and the search performed on PubMed resulted in 578 articles. After applying the Joanna Briggs Institute methodology for inclusion and exclusion, 24 cross-sectional studies were selected. Results: The selected articles were based on studies conducted between 1995 and 2021, in various continents and in both clinical and surgical residency programs, including outpatient, inpatient, and emergency settings. The Mini-CEX was shown to be applicable in the context of medical residency, as it is an observational assessment of the care provided by the resident physician in various practice settings such as outpatient clinics, inpatient wards, and emergency departments. It involves a variable observation time ranging from 10 to 40 minutes and allows for the evaluation of various aspects of medical care, including history taking, physical examination, clinical reasoning, counseling, and provides an opportunity for providing feedback on the residents' performance. Conclusion: The mini-CEX is a tool that is easy to implement and promotes a high degree of satisfaction among stakeholders. It could be used more routinely in medical residency programs.

Training first-year medical residents to break bad news using healthcare role-play and trainees as simulated patients: Experience of the "ADIAMED" program from Lille University School of Medicine.

INTRODUCTION: Several studies suggest the relevance of healthcare simulation to prepare future doctors to deliver bad news. A such, we designed a role-play workshop to train first-year residents enrolled in Lille University School of Medicine to break bad news. The objective of this work is to report on our experience of this training and to assess its educational value through its capacity to satisfy residents' expectations, to induce a feeling of ease towards bad news disclosure, and to change trainees' preconceptions regarding these situations. METHODS: The training consisted of a 45-minute heuristic reflective activity, aimed at identifying residents' preconceptions regarding bad news disclosure, followed by 4 30-min role-plays in which they played the parts of the physician, the patient and/or their relatives. Trainees were asked to answer 2 questionnaires (pre- and post-training), exploring previous experiences, preconceived ideas regarding bad news disclosure and workshop satisfaction. RESULTS: Almost all residents felt very satisfied with the workshop, which they regarded as formative (91%) and not too stressful (89%). The majority felt "more capable" (53% vs. 83%) and "more comfortable" (27% vs. 62%) to deliver bad news, especially regarding "finding the right words" (12% vs. 22%). Trainees tended to overestimate their skills before the workshop and lowered their assessment of their performance after attending the training, especially when they played the role of a patient in the simulation. CONCLUSION: Healthcare role-play seems an interesting technique for training to breaking bad news. Placing residents in the role of patients or relatives is an active approach that encourages reflexivity.

Reimagining a pass/fail clinical core clerkship: a US residency program director survey and meta-analysis.

Pass/fail (P/F) grading has emerged as an alternative to tiered clerkship grading. Systematically evaluating existing literature and surveying program directors (PD) perspectives on these consequential changes can guide educators in addressing inequalities in academia and students aiming to improve their residency applications. In our survey, a total of 1578 unique PD responses (63.1%) were obtained across 29 medical specialties. With the changes to United States Medical Licensure Examination (USMLE), responses showed increased importance of core clerkships with the implementation of Step 2CK cutoffs. PDs believed core clerkship performance was a reliable representation of an applicant's preparedness for residency, particularly in Accreditation Council for Graduate Medical Education's (ACGME)Medical Knowledge and Patient Care and Procedural Skills. PDs disagreed with P/F core clerkships because it more difficult to objectively compare applicants. No statistically significant differences in responses were found in PD preferential selection when comparing applicants from tiered and P/F core clerkship grading systems. If core clerkships adopted P/F scoring, PDs would further increase emphasis on narrative assessment, sub-internship evaluation, reference letters, academic awards, professional development and medical school prestige. In the meta-analysis, of 6 studies from 2,118 participants, adjusted scaled scores with mean difference from an equal variance model from PDs showed residents from tiered clerkship grading systems overall performance, learning ability, work habits, personal evaluations, residency selection and educational evaluation were not statistically significantly different than from residents from P/F systems. Overall, our dual study suggests that while PDs do not favor P/F core clerkships, PDs do not have a selection preference and do not report a difference in performance between applicants from P/F vs. tiered grading core clerkship systems, thus providing fertile grounds for institutions to examine the feasibility of adopting P/F grading for core clerkships.

e-Learning Evaluation Framework and Tools for Global Health and Public Health Education: Protocol for a Scoping Review.

BACKGROUND: There has been a significant increase in the use of e-learning for global and public health education recently, especially following the COVID-19 pandemic. e-Learning holds the potential to offer equal opportunities, overcoming barriers like physical limitations and training costs. However, its effectiveness remains debated, with institutions unprepared for the sudden shift during the pandemic. To effectively evaluate the outcomes of e-learning, a standardized and rigorous approach is necessary. However, the existing literature on this subject often lacks standardized assessment tools and theoretical foundations, leading to ambiguity in the evaluation process. Consequently, it becomes imperative to identify a clear theoretical foundation and practical approach for evaluating global and public health e-learning outcomes. OBJECTIVE: This protocol for a scoping review aims to map the state of e-learning evaluation in global and public health education to determine the existing theoretical evaluation frameworks, methods, tools, and domains and the gaps in research and practice. METHODS: The scoping review will be conducted following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) guidelines. The initial search was performed in PubMed, Education Resource Information Center, Web of Science, and Scopus to identify peer-reviewed articles that report on the use of evaluation and assessment for e-learning training. The search strings combined the concepts of e-learning, public health, and health science education, along with evaluation and frameworks. After the initial search, a screening process will be carried out to determine the relevance of the identified studies to the research question. Data related to the characteristics of the included studies, the characteristics of the e-learning technology used in the studies, and the study outcomes will be extracted from the eligible articles. The extracted data will then undergo a structured, descriptive, quantitative, and qualitative content analysis to synthesize the information from the selected studies. RESULTS:  Initial database searches yielded a total of 980 results. Duplicates have been removed, and title and abstract screening of the 805 remaining extracted articles are underway. Quantitative and qualitative findings from the reviewed articles will be presented to answer the study objective. CONCLUSIONS: This scoping review will provide global and public health educators with a comprehensive overview of the current state of e-learning evaluation. By identifying existing e-learning frameworks and tools, the findings will offer valuable guidance for further advancements in global and public health e-learning evaluation. The study will also enable the creation of a comprehensive, evidence-based e-learning evaluation framework and tools, which will improve the quality and accountability of global health and public health education. Ultimately, this will contribute to better health outcomes. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/49955.

Educating Family Physician Residents for Anesthetic Preoperative Evaluation and Assessment.

Introduction: Encounters for preoperative assessments are common within primary care offices, so it is imperative that family medicine residents learn how to perform preoperative evaluations. We assessed family medicine residents' knowledge of preoperative evaluation in preparation for surgery by providing a pre- and post-test alongside a didactic seminar. Methods: A didactic seminar on preoperative evaluations was presented at a family medicine resident didactics session by two senior anesthesiology residents. A 16-question, multiple choice test was used as both a pre-test and post-test to assess family medicine residents' knowledge. Results: A total of 31 participants took the pre-test (residents = 24; medical students = 7), and 30 participants took the post-test (residents = 23; medical students = 7). Mean scores and standard deviations were calculated for both tests with an average score of 37.50% ± 10.58% and 45.42% ± 11.12% on the pre- and post-test, respectively. Using the Kruskal-Wallis test, residents showed a significant improvement in test scores following the didactic presentation (p = 0.041), while overall results (residents and medical students) also reported a significant difference (p = 0.004). Conclusions: Our results demonstrated that educating family medicine residents and medical students on preoperative evaluation showed significant, quantifiable gains in knowledge following a brief didactic presentation. Given the current gap between guidelines and practice, our results emphasize the need for a formal medical school and residency-based curriculum related to preoperative patient evaluation.

Educational implications of assessing learning outcomes with multiple choice questions and short essay questions.

PURPOSE: This study investigates the characteristics of different item types to assess learning outcomes and explore the educational implications that can be obtained from the results of learning outcome assessments. METHODS: Forty-five second-year premedical students participated in this study. Multiple choice question (MCQ) and short essay question (SEQ) scores and pass rates for 10 learning outcomes were analyzed. Descriptive statistics and correlation analysis were used to analyze the data. RESULTS: The correlation analysis indicated that there was a significant correlation between SEQs and pass rate but there was no significant correlation between MCQs and pass rate. Some students with identical scores on the MCQs had different scores on the SEQs or on the learning outcomes. CONCLUSION: This study showed that students' achievement of learning outcomes can be assessed using various types of questions in outcome-based education.

Student-created vignettes to enhance pharmacy student knowledge and skill in managing patients with neuropsychiatric disease

Backgound: Neuropsychiatric disease is common globally. It is vital train pharmacists to provide patient-centered care in neuropsychiatry. Objective: To evaluate the impact of student-created vignettes on their knowledge and abilities to assess and manage patients with neuropsychiatric diseases, and to evaluate their experience. Methods: Several learning/assessment methodologies within the Therapeutics III course were utilized, including a major assignment of student-created vignettes about neuropsychiatric diseases. A framework guided student in creating the vignettes; identifying conception, design, and administration. Created vignettes were evaluated based on a validated scoring guide. Mean scores in various assessments were compared using Spearman’s rank-order correlation. Students evaluated their experience on a 5-point Likert-type scale of 1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly Agree. Results: Overall, students’ performance in the assignment was excellent, average score = 92%. A significant correlation existed between the vignette assignment and assessments covering neuropsychiatric disease. Most students agreed they were made aware of what needed to be done (95%), that the instructions about elements to include, designs, and delivery mechanisms were enough (93.4%, 86.7%, and 93.4%, respectively). Most students agreed that developing the vignette was stimulating, engaging and enjoyable (93.3% and 90%, 88.3% respectively). Students stated they felt confident in their scientific background knowledge (88.3%), in employing communication strategies with patients (85%) and their families (83.3%), and in their confidence in promoting and supporting patients with the diseases. (AU)

Measuring Cognitive and Social Interactive Attributes of Digital Natives: Development and Validation of a Scale.

Scholars refer to individuals who have been immersed in digital environments and who make easy use of digital languages to interact with the world as "digital natives," and Teo proposed four attributes of digital natives to illustrate their behavioral tendencies. We aimed to expand Teo's framework and to develop and validate the Scale of Digital Native Attributes (SDNA) for measuring cognitive and social interactive attributes of digital natives. Based on pre-test results, we retained 10 attributes and 37 SDNA items, with 3-4 items in each sub-dimension. We then recruited 887 Taiwanese undergraduates as respondents and conducted confirmatory factor analysis to establish construct validity. Moreover, the SDNA correlated with several other related measurements to demonstrate satisfactory criterion-related validity. Internal consistency was evaluated by McDonald's Omega (ω) and Cronbach's α coefficient, showing satisfactory reliability. This preliminary tool is now ready for cross validation and temporal reliability testing in further research.

A method for rewarding collaborative efforts in preclinical assessments.

Student success in basic medical science courses is typically determined by their individual performance on examinations of various types. Previous research both within and outside medical education has shown that the use of educational assessment activities can increase learning as demonstrated by performance on subsequent examinations, a phenomenon known as the testing effect. Activities primarily designed and used for assessment and evaluation purposes can also be used as teaching opportunities. We developed a method for measuring and evaluating student accomplishment in a preclinical basic science course that incorporates both individual and collaborative efforts, encourages and rewards active participation, does not compromise the reliability of the assessment outcome and is perceived by the students as helpful and valuable. The approach involved a two-part assessment activity composed of an individual examination and a small group examination with each component differentially weighted in determining an overall examination score. We found that the method was successful in encouraging collaborative efforts during the group component and provided valid measures of student grasp of the subject matter. We describe the development and implementation of the method, provide data derived from its use in a preclinical basic science course and discuss factors to be addressed when utilizing this approach to ensure fairness and reliability of the outcome. We include brief summary comments from students regarding their impressions of the value of this method.

[A therapeutic education program for patients with multiple sclerosis]. / Un programme d'éducation thérapeutique pour les patients atteints de sclérose en plaques.

The MOTIV-SEP therapeutic education program for people with multiple sclerosis integrates the cognitive, emotional, behavioral and social components of therapeutic compliance. These components allow for an approach centered on the individual, in order to help him or her better support the important stage of starting treatment, but also to anticipate and reduce the obstacles to compliance.

Scoring Single-Response Multiple-Choice Items: Scoping Review and Comparison of Different Scoring Methods.

BACKGROUND: Single-choice items (eg, best-answer items, alternate-choice items, single true-false items) are 1 type of multiple-choice items and have been used in examinations for over 100 years. At the end of every examination, the examinees' responses have to be analyzed and scored to derive information about examinees' true knowledge. OBJECTIVE: The aim of this paper is to compile scoring methods for individual single-choice items described in the literature. Furthermore, the metric expected chance score and the relation between examinees' true knowledge and expected scoring results (averaged percentage score) are analyzed. Besides, implications for potential pass marks to be used in examinations to test examinees for a predefined level of true knowledge are derived. METHODS: Scoring methods for individual single-choice items were extracted from various databases (ERIC, PsycInfo, Embase via Ovid, MEDLINE via PubMed) in September 2020. Eligible sources reported on scoring methods for individual single-choice items in written examinations including but not limited to medical education. Separately for items with n=2 answer options (eg, alternate-choice items, single true-false items) and best-answer items with n=5 answer options (eg, Type A items) and for each identified scoring method, the metric expected chance score and the expected scoring results as a function of examinees' true knowledge using fictitious examinations with 100 single-choice items were calculated. RESULTS: A total of 21 different scoring methods were identified from the 258 included sources, with varying consideration of correctly marked, omitted, and incorrectly marked items. Resulting credit varied between -3 and +1 credit points per item. For items with n=2 answer options, expected chance scores from random guessing ranged between -1 and +0.75 credit points. For items with n=5 answer options, expected chance scores ranged between -2.2 and +0.84 credit points. All scoring methods showed a linear relation between examinees' true knowledge and the expected scoring results. Depending on the scoring method used, examination results differed considerably: Expected scoring results from examinees with 50% true knowledge ranged between 0.0% (95% CI 0% to 0%) and 87.5% (95% CI 81.0% to 94.0%) for items with n=2 and between -60.0% (95% CI -60% to -60%) and 92.0% (95% CI 86.7% to 97.3%) for items with n=5. CONCLUSIONS: In examinations with single-choice items, the scoring result is not always equivalent to examinees' true knowledge. When interpreting examination scores and setting pass marks, the number of answer options per item must usually be taken into account in addition to the scoring method used.

The Relation Between Students' Theoretical Knowledge and Practical Skills in Endodontics: Retrospective Analysis.

BACKGROUND: Dental undergraduate students are required to show sufficient practical skills prior to treating patients. Practical skills and the underlying theoretical knowledge are taught in preclinical courses. Usually, the learning outcome is assessed in written multiple-choice examinations (theoretical knowledge) and practical skills tests. However, students' assessment of practical skills is more time consuming and prone to bias than objective multiple-choice examinations. OBJECTIVE: This study aims to analyze the relation between students' theoretical knowledge and practical skills in endodontics. Furthermore, the predictive validity of a theoretical knowledge assessment on students' practical skills was assessed. METHODS: Examination results from all students who participated in the preclinical phantom course in Operative Dentistry (sixth semester of the undergraduate dental curriculum in Germany) between the 2015 summer term and the 2022 summer term were retrospectively evaluated (N=447). The effects of age, sex, previous course participation, and theoretical knowledge on students' practical skills were assessed, using Pearson correlations, Wilcoxon rank sum tests, and a linear regression analysis. Subsequently, students' theoretical knowledge and practical skills were compared via a Fisher exact test to identify a suitable pass mark for students' theoretical knowledge that was associated with sufficient practical skills (≥60%). RESULTS: Students' theoretical knowledge was significantly associated with practical skills (Padjusted=.02; r=0.13). By using the current pass mark for theoretical knowledge (ie, 60%), a significant differentiation between insufficient practical skills (<60%) and sufficient practical skills (≥60%) was achieved (P=.02). However, for the discrimination between students with sufficient practical skills and students with insufficient practical skills, an adapted pass mark for theoretical knowledge would be more appropriate. The ideal pass mark amounted to 58% (P=.02). CONCLUSIONS: Students' practical skills and theoretical knowledge are significantly correlated. By objectively measuring students' theoretical knowledge, a rough estimation of students' practical skills (ie, a differentiation between sufficient and insufficient practical skills) is possible.

Ophthalmology residency training in Nigeria: The trainers' perspective.

Introduction: Periodic evaluations are an indispensable part of any training programme. This study assessed ophthalmology residency training in Nigeria from the perspective of ophthalmology trainers. Materials and Methods: This was a nationwide web-based survey of ophthalmology trainers at the 30 institutions accredited by the National Postgraduate Medical College of Nigeria from April 2021 to June 2021. Trainers were invited to complete a Google form via E-mail and WhatsApp. Data were collected on the characteristics of trainers and the training programme. Results: One hundred and fifty-eight (71.2%) out of 222 trainers responded to the survey comprising 67 (42.4%) males and 91 (57.6%) females, amongst whom were 97 (61.4%) subspecialists. The mean age (±standard deviation) was 51.47 ± 8.61 years (range, 38-75 years). Most respondents rated human resources (trainers and support staff) as 'much more' than adequate (n = 30, 19%), the volume of surgery as 'less or much less' than adequate (n = 82, 51.9%), the conduct of examinations as 'good' or better than good (n = 120, 75.9%) and impact of emigration of ophthalmologists on training as 'very negative' (n = 36, 22.8%). Overall, 94 (59.4%) respondents rated the quality of training as 'good' or better than good. Respondents recommended improving funding and training resources, revising the current conduct of examinations, increasing the use of appropriate technology and improving remunerations and national security. Conclusions: The majority of ophthalmology trainers rated ophthalmology residency training in Nigeria as 'good' but rated 'volume of surgery' as inadequate for training. Recommendations were made to improve the quality of training, revise examination practices and reduce the negative impact of emigration.

Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods.

BACKGROUND: Scoring and awarding credit are more complex for multiple-select items than for single-choice items. Forty-one different scoring methods were retrospectively applied to 2 multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF]) from existing examination data. OBJECTIVE: This study aimed to calculate and compare the mean scores for both item types by applying different scoring methods, and to investigate the effect of item quality on mean raw scores and the likelihood of resulting scores at or above the pass level (≥0.6). METHODS: Items and responses from examinees (ie, marking events) were retrieved from previous examinations. Different scoring methods were retrospectively applied to the existing examination data to calculate corresponding examination scores. In addition, item quality was assessed using a validated checklist. Statistical analysis was performed using the Kruskal-Wallis test, Wilcoxon rank-sum test, and multiple logistic regression analysis (P<.05). RESULTS: We analyzed 1931 marking events of 48 Pick-N items and 828 marking events of 18 MTF items. For both item types, scoring results widely differed between scoring methods (minimum: 0.02, maximum: 0.98; P<.001). Both the use of an inappropriate item type (34 items) and the presence of cues (30 items) impacted the scoring results. Inappropriately used Pick-N items resulted in lower mean raw scores (0.88 vs 0.93; P<.001), while inappropriately used MTF items resulted in higher mean raw scores (0.88 vs 0.85; P=.001). Mean raw scores were higher for MTF items with cues than for those without cues (0.91 vs 0.8; P<.001), while mean raw scores for Pick-N items with and without cues did not differ (0.89 vs 0.90; P=.09). Item quality also impacted the likelihood of resulting scores at or above the pass level (odds ratio ≤6.977). CONCLUSIONS: Educators should pay attention when using multiple-select multiple-choice items and select the most appropriate item type. Different item types, different scoring methods, and presence of cues are likely to impact examinees' scores and overall examination results.

Comparing the prediction performance of item response theory and machine learning methods on item responses for educational assessments.

To obtain more accurate and robust feedback information from the students' assessment outcomes and to communicate it to students and optimize teaching and learning strategies, educational researchers and practitioners must critically reflect on whether the existing methods of data analytics are capable of retrieving the information provided in the database. This study compared and contrasted the prediction performance of an item response theory method, particularly the use of an explanatory item response model (EIRM), and six supervised machine learning (ML) methods for predicting students' item responses in educational assessments, considering student- and item-related background information. Each of seven prediction methods was evaluated through cross-validation approaches under three prediction scenarios: (a) unrealized responses of new students to existing items, (b) unrealized responses of existing students to new items, and (c) missing responses of existing students to existing items. The results of a simulation study and two real-life assessment data examples showed that employing student- and item-related background information in addition to the item response data substantially increases the prediction accuracy for new students or items. We also found that the EIRM is as competitive as the best performing ML methods in predicting the student performance outcomes for the educational assessment datasets.

Association Between Accreditation Era, North American Pharmacist Licensure Examination Testing Changes, and First-Time Pass Rates.

Objective. To estimate whether first-time pass rates on the North American Pharmacist Licensure Examination (NAPLEX) have been influenced by the number of pharmacy programs founded since 2000, the programs' accreditation era, and the changes to the blueprint as well as changes to the testing conditions and passing standards implemented by the National Association of Boards of Pharmacy (NABP) beginning in 2015.Methods. This was a retrospective, observational cohort study using publicly published data. The number of programs and pass rates were collected from 2008 to 2020. Programs reporting pass rates from 2016 to 2020 were eligible. Accreditation era was defined as programs accredited before or after 2000. Pass rates were categorized into NAPLEX tests administered before or after 2015. Statistical analyses were conducted for comparisons.Results. Pass rates were initially found to decline as the number of programs rose. First-time pass rates of programs accredited before 2000 were higher than pass rates of programs accredited after 2000 every year after 2011. Only 40% of the programs accredited after 2000 exceeded the national average between 2016-2020. Blueprint changes implemented in 2015 and the changes to testing conditions plus passing standards implemented in 2016 had a greater effect on pass rates than the number of programs or applicants.Conclusion. Programs accredited after 2000 generally had lower first-time NAPLEX pass rates. Even so, blueprint changes and changes to the testing conditions plus passing standards instituted by the NABP were more important predictors of the decline of first-time NAPLEX pass rates. Stakeholders should collaborate and embrace best practices for assessing practice-ready competency for licensure.